Evaluation of p-curve simulation results

Author

Felix Schönbrodt & Angelika Stefan

Published

September 23, 2025

In the following plots, always the RMSE goodness-of-fit measure is used (not chi2).

Some synthetic p-curves: Testing specific scenarios

Test a pure bump

Multiple DVs

Caution

I thought, the multiDV cannot produce a bump? Is this still MC error, or is there a real bump?

The best fitting scenarios from the “multiple DVs” simulations are:

Code
simres |> filter(type == "multDV") |> arrange(rmseBump) |> slice(1:5) |> select(c(1:9, 10:14), rmseBump) |> round(2) |> print()
  nvar   r   d prop_Hacker prop_H1 het  iter alpha strategy   p1   p2   p3   p4
1  150 0.9 0.2        0.75     0.1 0.4 10000  0.05        3 0.14 0.11 0.15 0.23
2   70 0.9 0.4        1.00     0.1 0.2 10000  0.05        3 0.15 0.11 0.15 0.23
3  150 0.9 0.2        1.00     0.5 0.0 10000  0.05        3 0.15 0.10 0.16 0.23
4  150 0.9 0.2        1.00     0.2 0.4 10000  0.05        3 0.18 0.10 0.15 0.22
5  150 0.9 0.3        0.75     0.2 0.2 10000  0.05        3 0.20 0.11 0.14 0.21
    p5 rmseBump
1 0.37     0.04
2 0.36     0.04
3 0.36     0.05
4 0.35     0.05
5 0.34     0.05

Optional Stopping

Code
# all optStop scenarios:
plot_pcurves(simres |> filter(type == "optStop"), poriginal = reference, n_best = 10) + ggtitle("Optional Stopping")

The best fitting scenarios from the “optional stopping” simulations are:

Code
simres |> filter(type == "optStop") |> arrange(rmseBump) |> slice(1:5) |> select(c(18:20, 10:14), rmseBump) |> round(2) |> print()
  nmin nmax stepsize   p1   p2   p3   p4   p5 rmseBump
1  444 1063        1 0.16 0.15 0.15 0.15 0.39     0.01
2   50  444        1 0.15 0.13 0.16 0.16 0.40     0.01
3  177 1063        1 0.15 0.14 0.14 0.14 0.43     0.01
4  177 1063        5 0.15 0.14 0.14 0.18 0.39     0.02
5  177  444        1 0.17 0.15 0.16 0.14 0.37     0.02

Test a “true signal + bump” pattern

Multiple DVs

Code
reference2 <- c(.40, .20, .10, .05, .25)
#reference <- c(.45, .22, .14, .05, .14)

# all multDV scenarios:
simres$rmseBump2 <- rmse(reference2, simres[, paste0("p", 1:5)])
plot_pcurves(simres |> filter(type == "multDV"), poriginal = reference2, n_best = 10, GOF="rmse")

The best fitting scenarios from the “multiple DVs” simulations are:

Code
simres |> filter(type == "multDV") |> arrange(rmseBump2) |> slice(1:5) |> select(c(1:9, 10:14), rmseBump2) |> round(2) |> print()
  nvar   r   d prop_Hacker prop_H1 het  iter alpha strategy   p1   p2   p3   p4
1   10 0.9 0.4        0.10     0.1 0.0 10000  0.05        1 0.38 0.20 0.15 0.12
2   10 0.9 0.4        0.10     0.1 0.0 10000  0.05        2 0.38 0.20 0.15 0.12
3    2 0.0 0.4        0.25     0.1 0.2 10000  0.05        1 0.41 0.18 0.13 0.13
4    2 0.0 0.4        0.25     0.1 0.2 10000  0.05        2 0.41 0.18 0.13 0.13
5    5 0.6 0.5        0.50     0.1 0.2 10000  0.05        1 0.40 0.18 0.14 0.12
    p5 rmseBump2
1 0.16      0.06
2 0.16      0.06
3 0.16      0.06
4 0.16      0.06
5 0.15      0.06

Optional Stopping

Code
# all optStop scenarios:
plot_pcurves(simres |> filter(type == "optStop"), poriginal = reference2, n_best = 10, GOF="rmse")

The best fitting scenarios from the “optional stopping” simulations are:

Code
simres |> filter(type == "optStop") |> arrange(rmseBump2) |> slice(1:5) |> select(c(18:20, 10:14), rmseBump2) |> round(2) |> print()
  nmin nmax stepsize   p1   p2   p3   p4   p5 rmseBump2
1  444 1063        1 0.43 0.14 0.10 0.08 0.26      0.03
2  177  444        1 0.40 0.14 0.13 0.09 0.24      0.04
3  444 1063        1 0.38 0.15 0.11 0.11 0.25      0.04
4  177 1063        1 0.40 0.14 0.11 0.10 0.25      0.04
5  177  444        1 0.41 0.15 0.11 0.11 0.22      0.04

Three empirical p-curves

“Sotola”

Perfect set (no p-hacking, real effects)

Code
plot_pcurves(simres |> filter(condition == "multDV_perfect"), poriginal = pcurves[pcurves$dataset == "sotola", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV perfect")

Code
simres |> filter(condition == "multDV_perfect") |> arrange(rmseSotola) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSotola) |> round(3) |> print()
  nvar r   d prop_Hacker prop_H1 het  iter alpha strategy nmin nmax stepsize
1    1 0 0.8           0     0.2 0.2 10000  0.05        1   NA   NA       NA
2    1 0 0.8           0     0.2 0.2 10000  0.05        2   NA   NA       NA
3    1 0 0.8           0     0.2 0.2 10000  0.05        3   NA   NA       NA
4    1 0 0.3           0     0.7 0.4 10000  0.05        1   NA   NA       NA
5    1 0 0.3           0     0.7 0.4 10000  0.05        2   NA   NA       NA
     p1    p2    p3    p4    p5 rmseSotola
1 0.666 0.105 0.092 0.064 0.074      0.024
2 0.666 0.105 0.092 0.064 0.074      0.024
3 0.666 0.105 0.092 0.064 0.074      0.024
4 0.651 0.120 0.088 0.066 0.075      0.028
5 0.651 0.120 0.088 0.066 0.075      0.028

H0 set (no p-hacking)

Code
plot_pcurves(simres |> filter(condition == "multDV_H0"), poriginal = pcurves[pcurves$dataset == "sotola", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV H0")
Warning: Removed 35 rows containing missing values or values outside the scale range
(`geom_line()`).

Code
simres |> filter(condition == "multDV_H0") |> arrange(rmseSotola) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSotola) |> round(3) |> print()
  nvar r d prop_Hacker prop_H1 het  iter alpha strategy nmin nmax stepsize
1    1 0 0           0       0   0 10000  0.05        1   NA   NA       NA
2    1 0 0           0       0   0 10000  0.05        2   NA   NA       NA
3    1 0 0           0       0   0 10000  0.05        3   NA   NA       NA
     p1    p2    p3    p4    p5 rmseSotola
1 0.211 0.209 0.183 0.217 0.179      0.226
2 0.211 0.209 0.183 0.217 0.179      0.226
3 0.211 0.209 0.183 0.217 0.179      0.226

multDV realistic set

Code
plot_pcurves(simres |> filter(condition == "multDV_realistic"), poriginal = pcurves[pcurves$dataset == "sotola", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV realistic")

Code
simres |> filter(condition == "multDV_realistic") |> arrange(rmseSotola) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSotola) |> round(3) |> print()
  nvar   r   d prop_Hacker prop_H1 het  iter alpha strategy nmin nmax stepsize
1    5 0.9 0.8        0.50     0.2 0.2 10000  0.05        1   NA   NA       NA
2    5 0.9 0.8        0.50     0.2 0.2 10000  0.05        2   NA   NA       NA
3    5 0.3 0.6        0.10     0.3 0.2 10000  0.05        1   NA   NA       NA
4    5 0.3 0.6        0.10     0.3 0.2 10000  0.05        2   NA   NA       NA
5    5 0.9 0.8        0.75     0.5 0.0 10000  0.05        3   NA   NA       NA
     p1    p2    p3    p4    p5 rmseSotola
1 0.643 0.107 0.099 0.066 0.084      0.022
2 0.643 0.107 0.099 0.066 0.084      0.022
3 0.635 0.113 0.107 0.065 0.081      0.023
4 0.635 0.113 0.107 0.065 0.081      0.023
5 0.662 0.102 0.091 0.069 0.076      0.025

multDV worst set

Code
plot_pcurves(simres |> filter(condition == "multDV_worst"), poriginal = pcurves[pcurves$dataset == "sotola", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV worst")

Code
simres |> filter(condition == "multDV_worst") |> arrange(rmseSotola) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSotola) |> round(3) |> print()
  nvar r d prop_Hacker prop_H1 het  iter alpha strategy nmin nmax stepsize
1  150 0 0        0.10       0   0 10000  0.05        1   NA   NA       NA
2  150 0 0        0.10       0   0 10000  0.05        2   NA   NA       NA
3  150 0 0        0.25       0   0 10000  0.05        1   NA   NA       NA
4  150 0 0        0.25       0   0 10000  0.05        2   NA   NA       NA
5  150 0 0        0.50       0   0 10000  0.05        1   NA   NA       NA
     p1    p2    p3    p4    p5 rmseSotola
1 0.596 0.182 0.091 0.076 0.055      0.059
2 0.596 0.182 0.091 0.076 0.055      0.059
3 0.701 0.176 0.056 0.038 0.028      0.062
4 0.701 0.176 0.056 0.038 0.028      0.062
5 0.747 0.179 0.045 0.019 0.010      0.078

optStop realistic set

Code
plot_pcurves(simres |> filter(condition == "optStop_realistic"), poriginal = pcurves[pcurves$dataset == "sotola", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola optStop realistic")

Code
simres |> filter(condition == "optStop_realistic") |> arrange(rmseSotola) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSotola) |> round(3) |> print()
  nvar  r   d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1   NA NA 0.6        0.10     0.3 0.2   NA    NA       NA  177 1063       50
2   NA NA 0.7        0.25     0.2 0.2   NA    NA       NA  444 1063      100
3   NA NA 0.8        0.10     0.2 0.0   NA    NA       NA   50 1063       50
4   NA NA 0.5        0.25     0.3 0.4   NA    NA       NA  444 1063      100
5   NA NA 0.5        0.25     0.3 0.4   NA    NA       NA  177  444       50
     p1    p2    p3    p4    p5 rmseSotola
1 0.660 0.099 0.099 0.067 0.075      0.021
2 0.663 0.092 0.089 0.069 0.088      0.023
3 0.643 0.100 0.098 0.076 0.084      0.024
4 0.670 0.091 0.085 0.067 0.088      0.025
5 0.656 0.103 0.092 0.072 0.077      0.025

optStop worst set

Code
plot_pcurves(simres |> filter(condition == "optStop_worst"), poriginal = pcurves[pcurves$dataset == "sotola", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola optStop worst")

Code
simres |> filter(condition == "optStop_worst") |> arrange(rmseSotola) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSotola) |> round(3) |> print()
  nvar  r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1   NA NA 0         0.1       0   0   NA    NA       NA   50  177      100
2   NA NA 0         0.1       0   0   NA    NA       NA   22  444      100
3   NA NA 0         0.1       0   0   NA    NA       NA  177  444       50
4   NA NA 0         0.1       0   0   NA    NA       NA  177  444      100
5   NA NA 0         0.1       0   0   NA    NA       NA   50  444      100
     p1    p2    p3    p4    p5 rmseSotola
1 0.227 0.204 0.162 0.204 0.204      0.219
2 0.209 0.179 0.206 0.175 0.232      0.225
3 0.208 0.172 0.222 0.215 0.182      0.227
4 0.206 0.206 0.186 0.189 0.212      0.227
5 0.210 0.205 0.184 0.220 0.181      0.227

“Wetzels”

Perfect set (no p-hacking, real effects)

Code
plot_pcurves(simres |> filter(condition == "multDV_perfect"), poriginal = pcurves[pcurves$dataset == "wetzels", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV perfect")

Code
simres |> filter(condition == "multDV_perfect") |> arrange(rmseWetzels) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseWetzels) |> round(3) |> print()
  nvar r   d prop_Hacker prop_H1 het  iter alpha strategy nmin nmax stepsize
1    1 0 0.5           0     0.5 0.4 10000  0.05        1   NA   NA       NA
2    1 0 0.5           0     0.5 0.4 10000  0.05        2   NA   NA       NA
3    1 0 0.5           0     0.5 0.4 10000  0.05        3   NA   NA       NA
4    1 0 0.7           0     0.3 0.2 10000  0.05        1   NA   NA       NA
5    1 0 0.7           0     0.3 0.2 10000  0.05        2   NA   NA       NA
     p1    p2    p3    p4    p5 rmseWetzels
1 0.688 0.107 0.074 0.076 0.055       0.004
2 0.688 0.107 0.074 0.076 0.055       0.004
3 0.688 0.107 0.074 0.076 0.055       0.004
4 0.686 0.112 0.081 0.065 0.057       0.006
5 0.686 0.112 0.081 0.065 0.057       0.006

H0 set (no p-hacking)

Code
plot_pcurves(simres |> filter(condition == "multDV_H0"), poriginal = pcurves[pcurves$dataset == "wetzels", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV H0")
Warning: Removed 35 rows containing missing values or values outside the scale range
(`geom_line()`).

Code
simres |> filter(condition == "multDV_H0") |> arrange(rmseWetzels) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseWetzels) |> round(3) |> print()
  nvar r d prop_Hacker prop_H1 het  iter alpha strategy nmin nmax stepsize
1    1 0 0           0       0   0 10000  0.05        1   NA   NA       NA
2    1 0 0           0       0   0 10000  0.05        2   NA   NA       NA
3    1 0 0           0       0   0 10000  0.05        3   NA   NA       NA
     p1    p2    p3    p4    p5 rmseWetzels
1 0.211 0.209 0.183 0.217 0.179       0.238
2 0.211 0.209 0.183 0.217 0.179       0.238
3 0.211 0.209 0.183 0.217 0.179       0.238

multDV realistic set

Code
plot_pcurves(simres |> filter(condition == "multDV_realistic"), poriginal = pcurves[pcurves$dataset == "wetzels", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV realistic")

Code
simres |> filter(condition == "multDV_realistic") |> arrange(rmseWetzels) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseWetzels) |> round(3) |> print()
  nvar   r   d prop_Hacker prop_H1 het  iter alpha strategy nmin nmax stepsize
1    2 0.0 0.7        0.50     0.5 0.0 10000  0.05        3   NA   NA       NA
2    2 0.9 0.6        0.25     0.5 0.2 10000  0.05        1   NA   NA       NA
3    2 0.9 0.6        0.25     0.5 0.2 10000  0.05        2   NA   NA       NA
4   10 0.0 0.6        0.75     0.5 0.2 10000  0.05        1   NA   NA       NA
5   10 0.0 0.6        0.75     0.5 0.2 10000  0.05        2   NA   NA       NA
     p1    p2    p3    p4    p5 rmseWetzels
1 0.690 0.112 0.074 0.072 0.052       0.002
2 0.684 0.117 0.074 0.072 0.054       0.002
3 0.684 0.117 0.074 0.072 0.054       0.002
4 0.688 0.116 0.076 0.070 0.049       0.002
5 0.688 0.116 0.076 0.070 0.049       0.002

multDV worst set

Code
plot_pcurves(simres |> filter(condition == "multDV_worst"), poriginal = pcurves[pcurves$dataset == "wetzels", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV worst")

Code
simres |> filter(condition == "multDV_worst") |> arrange(rmseWetzels) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseWetzels) |> round(3) |> print()
  nvar r d prop_Hacker prop_H1 het  iter alpha strategy nmin nmax stepsize
1  150 0 0        0.25       0   0 10000  0.05        1   NA   NA       NA
2  150 0 0        0.25       0   0 10000  0.05        2   NA   NA       NA
3  150 0 0        0.10       0   0 10000  0.05        1   NA   NA       NA
4  150 0 0        0.10       0   0 10000  0.05        2   NA   NA       NA
5  150 0 0        0.50       0   0 10000  0.05        1   NA   NA       NA
     p1    p2    p3    p4    p5 rmseWetzels
1 0.701 0.176 0.056 0.038 0.028       0.035
2 0.701 0.176 0.056 0.038 0.028       0.035
3 0.596 0.182 0.091 0.076 0.055       0.051
4 0.596 0.182 0.091 0.076 0.055       0.051
5 0.747 0.179 0.045 0.019 0.010       0.052

optStop realistic set

Code
plot_pcurves(simres |> filter(condition == "optStop_realistic"), poriginal = pcurves[pcurves$dataset == "wetzels", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola optStop realistic")

Code
simres |> filter(condition == "optStop_realistic") |> arrange(rmseWetzels) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseWetzels) |> round(3) |> print()
  nvar  r   d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1   NA NA 0.6        0.75     0.7 0.4   NA    NA       NA   22  177      100
2   NA NA 0.5        0.50     0.5 0.0   NA    NA       NA    8  177      100
3   NA NA 0.6        0.50     0.5 0.4   NA    NA       NA   50  177       50
4   NA NA 0.6        0.25     0.7 0.4   NA    NA       NA   22 1063      100
5   NA NA 0.8        0.10     0.3 0.4   NA    NA       NA   22  177       50
     p1    p2    p3    p4    p5 rmseWetzels
1 0.687 0.115 0.075 0.069 0.054       0.003
2 0.682 0.115 0.076 0.072 0.055       0.003
3 0.687 0.113 0.070 0.073 0.057       0.004
4 0.687 0.114 0.076 0.068 0.055       0.004
5 0.687 0.109 0.071 0.076 0.056       0.004

optStop worst set

Code
plot_pcurves(simres |> filter(condition == "optStop_worst"), poriginal = pcurves[pcurves$dataset == "wetzels", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola optStop worst")

Code
simres |> filter(condition == "optStop_worst") |> arrange(rmseWetzels) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseWetzels) |> round(3) |> print()
  nvar  r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1   NA NA 0        0.10       0   0   NA    NA       NA   50  177      100
2   NA NA 0        0.10       0   0   NA    NA       NA   50  444      100
3   NA NA 0        0.25       0   0   NA    NA       NA   22  177      100
4   NA NA 0        0.10       0   0   NA    NA       NA  177  444      100
5   NA NA 0        0.10       0   0   NA    NA       NA  177  444       50
     p1    p2    p3    p4    p5 rmseWetzels
1 0.227 0.204 0.162 0.204 0.204       0.231
2 0.210 0.205 0.184 0.220 0.181       0.239
3 0.207 0.200 0.195 0.207 0.191       0.240
4 0.206 0.206 0.186 0.189 0.212       0.241
5 0.208 0.172 0.222 0.215 0.182       0.241

“Simonsohn”

Perfect set (no p-hacking, real effects)

Code
plot_pcurves(simres |> filter(condition == "multDV_perfect"), poriginal = pcurves[pcurves$dataset == "simonsohn", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV perfect")

Code
simres |> filter(condition == "multDV_perfect") |> arrange(rmseSimonsohn) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSimonsohn) |> round(3) |> print()
  nvar r   d prop_Hacker prop_H1 het  iter alpha strategy nmin nmax stepsize
1    1 0 0.3           0     0.3 0.2 10000  0.05        1   NA   NA       NA
2    1 0 0.3           0     0.3 0.2 10000  0.05        2   NA   NA       NA
3    1 0 0.3           0     0.3 0.2 10000  0.05        3   NA   NA       NA
4    1 0 0.2           0     0.7 0.0 10000  0.05        1   NA   NA       NA
5    1 0 0.2           0     0.7 0.0 10000  0.05        2   NA   NA       NA
     p1    p2    p3    p4    p5 rmseSimonsohn
1 0.474 0.169 0.137 0.102 0.118         0.038
2 0.474 0.169 0.137 0.102 0.118         0.038
3 0.474 0.169 0.137 0.102 0.118         0.038
4 0.427 0.188 0.145 0.122 0.118         0.041
5 0.427 0.188 0.145 0.122 0.118         0.041

H0 set (no p-hacking)

Code
plot_pcurves(simres |> filter(condition == "multDV_H0"), poriginal = pcurves[pcurves$dataset == "simonsohn", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV H0")
Warning: Removed 35 rows containing missing values or values outside the scale range
(`geom_line()`).

Code
simres |> filter(condition == "multDV_H0") |> arrange(rmseSimonsohn) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSimonsohn) |> round(3) |> print()
  nvar r d prop_Hacker prop_H1 het  iter alpha strategy nmin nmax stepsize
1    1 0 0           0       0   0 10000  0.05        1   NA   NA       NA
2    1 0 0           0       0   0 10000  0.05        2   NA   NA       NA
3    1 0 0           0       0   0 10000  0.05        3   NA   NA       NA
     p1    p2    p3    p4    p5 rmseSimonsohn
1 0.211 0.209 0.183 0.217 0.179         0.136
2 0.211 0.209 0.183 0.217 0.179         0.136
3 0.211 0.209 0.183 0.217 0.179         0.136

multDV realistic set

Code
plot_pcurves(simres |> filter(condition == "multDV_realistic"), poriginal = pcurves[pcurves$dataset == "simonsohn", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV realistic")

Code
simres |> filter(condition == "multDV_realistic") |> arrange(rmseSimonsohn) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSimonsohn) |> round(3) |> print()
  nvar   r   d prop_Hacker prop_H1 het  iter alpha strategy nmin nmax stepsize
1   70 0.3 0.2        0.10     0.3 0.2 10000  0.05        1   NA   NA       NA
2   70 0.3 0.2        0.10     0.3 0.2 10000  0.05        2   NA   NA       NA
3   70 0.0 0.1        0.10     0.1 0.2 10000  0.05        1   NA   NA       NA
4   70 0.0 0.1        0.10     0.1 0.2 10000  0.05        2   NA   NA       NA
5   70 0.3 0.1        0.25     0.7 0.0 10000  0.05        1   NA   NA       NA
     p1    p2    p3    p4    p5 rmseSimonsohn
1 0.451 0.207 0.136 0.103 0.103         0.031
2 0.451 0.207 0.136 0.103 0.103         0.031
3 0.462 0.226 0.142 0.088 0.082         0.031
4 0.462 0.226 0.142 0.088 0.082         0.031
5 0.475 0.209 0.149 0.083 0.085         0.031

multDV worst set

Code
plot_pcurves(simres |> filter(condition == "multDV_worst"), poriginal = pcurves[pcurves$dataset == "simonsohn", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV worst")

Code
simres |> filter(condition == "multDV_worst") |> arrange(rmseSimonsohn) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSimonsohn) |> round(3) |> print()
  nvar   r d prop_Hacker prop_H1 het  iter alpha strategy nmin nmax stepsize
1  150 0.3 0         0.1       0   0 10000  0.05        1   NA   NA       NA
2  150 0.3 0         0.1       0   0 10000  0.05        2   NA   NA       NA
3   70 0.0 0         0.1       0   0 10000  0.05        1   NA   NA       NA
4   70 0.0 0         0.1       0   0 10000  0.05        2   NA   NA       NA
5   70 0.3 0         1.0       0   0 10000  0.05        1   NA   NA       NA
     p1    p2    p3    p4    p5 rmseSimonsohn
1 0.459 0.207 0.145 0.107 0.082         0.038
2 0.459 0.207 0.145 0.107 0.082         0.038
3 0.412 0.263 0.143 0.102 0.080         0.044
4 0.412 0.263 0.143 0.102 0.080         0.044
5 0.431 0.235 0.155 0.110 0.069         0.044

optStop realistic set

Code
plot_pcurves(simres |> filter(condition == "optStop_realistic"), poriginal = pcurves[pcurves$dataset == "simonsohn", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola optStop realistic")

Code
simres |> filter(condition == "optStop_realistic") |> arrange(rmseSimonsohn) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSimonsohn) |> round(3) |> print()
  nvar  r   d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1   NA NA 0.4        0.25     0.2 0.0   NA    NA       NA    8  177      100
2   NA NA 0.4        0.10     0.5 0.0   NA    NA       NA    8  177       50
3   NA NA 0.2        0.10     0.7 0.2   NA    NA       NA   50  444      100
4   NA NA 0.4        0.50     0.5 0.0   NA    NA       NA    8  177       50
5   NA NA 0.3        0.25     0.7 0.0   NA    NA       NA   50  177      100
     p1    p2    p3    p4    p5 rmseSimonsohn
1 0.461 0.184 0.133 0.103 0.119         0.033
2 0.474 0.188 0.135 0.098 0.106         0.034
3 0.474 0.180 0.140 0.097 0.110         0.035
4 0.466 0.191 0.134 0.105 0.104         0.035
5 0.437 0.193 0.149 0.107 0.113         0.035

optStop worst set

Code
plot_pcurves(simres |> filter(condition == "optStop_worst"), poriginal = pcurves[pcurves$dataset == "simonsohn", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola optStop worst")

Code
simres |> filter(condition == "optStop_worst") |> arrange(rmseSimonsohn) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSimonsohn) |> round(3) |> print()
  nvar  r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1   NA NA 0        0.10       0   0   NA    NA       NA   50  177      100
2   NA NA 0        0.10       0   0   NA    NA       NA  177  444      100
3   NA NA 0        0.10       0   0   NA    NA       NA   22  444      100
4   NA NA 0        0.50       0   0   NA    NA       NA    8  177      100
5   NA NA 0        0.25       0   0   NA    NA       NA   22  177      100
     p1    p2    p3    p4    p5 rmseSimonsohn
1 0.227 0.204 0.162 0.204 0.204         0.128
2 0.206 0.206 0.186 0.189 0.212         0.135
3 0.209 0.179 0.206 0.175 0.232         0.137
4 0.200 0.212 0.185 0.192 0.212         0.137
5 0.207 0.200 0.195 0.207 0.191         0.138